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ERGODIC DIFFUSION CONTROL OF MULTICLASS MULTI-POOL 
NETWORKS IN THE HALFIN-WHITT REGIME 
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Abstract. We consider Markovian multiclass multi-pool networks with heterogeneous server pools, 
each consisting of many statistically identical parallel servers, where the bipartite graph of customer 
classes and server pools forms a tree. Customers form their own queue and are served in the first- 
come first-served discipline, and can abandon while waiting in queue. Service rates are both class 
and pool dependent. The objective is to study the limiting diffusion control problems under the 
long run average (ergodic) cost criteria in the Halfin-Whitt regime. Two formulations of ergodic 
diffusion control problems are considered: (i) both queueing and idleness costs are minimized, and 
(ii) only the queueing cost is minimized while a constraint is imposed upon the idleness of all server 
pools. We develop a recursive leaf elimination algorithm that enables us to obtain an explicit rep¬ 
resentation of the drift for the controlled diffusions. Consequently, we show that for the limiting 
controlled diffusions, there always exists a stationary Markov control under which the diffusion pro¬ 
cess is geometrically ergodic. The framework developed in pQ is extended to address a broad class 
of ergodic diffusion control problems with constraints. We show that that the unconstrained and 
constrained problems are well posed, and we characterize the optimal stationary Markov controls 
via HJB equations. 


1. Introduction 

Consider a multiclass parallel server networks with I classes of customers (jobs) and J parallel 
server pools, each of which has many statistically identical servers. Customers of each class can be 
served in a subset of the server pools, and each server pool can serve a subset of the customer classes, 
which forms a bipartite graph. We assume that this bipartite graph is a tree. Customers of each 
class arrive according to a Poisson process and form their own queue. They are served in the first- 
come-hrst-served (FCFS) discipline. Customers waiting in queue may renege if their patience times 
are reached before entering service. The patience times are exponentially distributed with class- 
dependent rates, while the service times are also exponentially distributed with rates depending on 
both the customer class and the server pool. The scheduling and routing control decides which class 
of customers to serve (if any waiting in queue) when a server becomes free, and which server pool to 
route a customer when multiple server pools have free servers to serve the customer. We focus on 
preemptive scheduling policies that satisfy the usual work conserving condition (no server can idle if 
a customer it can serve is in queue), as well as the joint work conserving condition [SHT] under which, 
customers can be rearranged in such a manner that no server will idle when a customer of some 
class is waiting in queue. In this paper, we study the diffusion control problems of such multiclass 
multi-pool networks under the long run average (ergodic) cost criteria in the Halfin-Whitt regime. 

We consider two formulations of the ergodic diffusion control problems. In the first formula¬ 
tion, both queueing and idleness are penalized in the running cost, and we refer to this as the 
“unconstrained” problem. In the second formulation, only the queueing cost is minimized, while a 
constraint is imposed upon the idleness of all server pools. We refer to this as the “constrained” 
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problem. The constraint can be regarded as a “fairness” condition on server pools. We aim to 
study the recurrence properties of the controlled diffusions, the well-posedness of these two ergodic 
diffusion control problems, and characterize the optimal stationary Markov controls via Hamilton- 
Jacobi-Bellman (HJB) equations. 

The diffusion limit of the queueing processes for the multiclass multi-pool networks was estab¬ 
lished by Atar mi- Certain properties of the controlled diffusions were proved in m, with the 
objective of studying the diffusion control problem under the discounted cost criterion. However 
those properties do not suffice for the study the ergodic control problem. Our first task is to ob¬ 
tain a good understanding of the recurrence properties of the limiting controlled diffusions. The 
main obstacle lies in the implicitness of the drift, which is represented via the solution of a linear 
program (Section 12.3p . Our first key contribution is to provide an explicit representation of the 
drift of the limiting controlled diffusions via a recursive leaf elimination algorithm (Sections 14.11 
and 14.2p . As a consequence, we show that the controlled diffusions have a piecewise linear drift 
fLemma 14.3p . which, unfortunately, does not belong to the class of piecewise linear diffusions stud¬ 
ied in m and [T], despite the somewhat similar representations. The dominating matrix in the 
drift is a Hurwitz lower-diagonal matrix, instead of the negative of an M-matrix. Applying the 
leaf elimination algorithm, we show that for any Markovian multiclass multi-pool (acyclic) network 
in the Halhn-Whitt regime, assuming that the abandonment rates are not identically zero, there 
exists a stationary Markov control under which the limiting diffusion is geometrically ergodic, and 
as a result, its invariant probability distribution has all moments hnite (Theorem 14.2p . 

A new framework to study ergodic diffusion control problems was introduced in [T], in order to 
study the multiclass single-pool network (the “V” model) in the Halhn-Whitt regime. It imposes 
a structural assumption (Hypothesis 3.1), which extends the applicability of the theory beyond 
the two dominant models in the study of ergodic control for diffusions [2]: (i) the running cost 
is near-monotone and (ii) the controlled diffusion is uniformly stable. The relevant results are 
reviewed in Section [3.21 Like the “V” model, the ergodic control problems of diffusions associated 
with multiclass multi-pool networks do not fall into any of those two categories. We show that 
the “unconstrained” ergodic diffusion control problem is well-posed and can be solved using the 
framework in [T]. Verihcation of the structural assumption in Hypothesis 3.1, relies heavily upon 
the explicit representation of the drift in the limiting controlled diffusions (Theorem 14.ip . We then 
establish the existence of an optimal stationary Markov control, characterize all such controls via 
an HJB equation in Section 15.21 

Ergodic control with constraints for diffusions was studied in mm-, see Sections 4.2 and 4.5 
in [2]. However, the existing methods and theory also fall into the same two categories mentioned 
above. Therefore, to study the well-posedness and solutions of the “constrained” problem, we 
extend the framework in [T] to ergodic diffusion control problems with constraints under the same 
structural assumption in Section 13.31 The well-posedness of the constrained problem follows by 
Lemma 13.31 of that section. We also characterize the optimal stationary controls via an HJB 
equation, which has a unique solution in a certain class (Theorems 13.11 and 13.2|) . We also extend 
the “spatial truncation” technique developed in [T] to problems under constraints (Theorems 13.31 
and 13.4p . These results are applied to the ergodic diffusion control problem with constraints for 
the multiclass multi-pool networks in Section 15.31 The special case of fair allocation of idleness in 
the constrained problem is discussed in Section [5.41 

It is worth noting that if we only penalize the queue but not the idleness, the unconstrained er¬ 
godic control problem may not be well-posed. We discuss the verification of the structural assump¬ 
tion (Hypothesis 3.1), in this formulation of the ergodic diffusion control problem in Section [4.41 
We show that under certain restrictions on the systems parameters or network structure. Hypothe¬ 
sis 3.1 can be verified and this formulation is therefore well-posed (see Corollaries 14.11 and 14.21 and 
Remark 14.6p . 
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1.1. Literature review. Scheduling and routing control of multiclass multi-pool networks in the 
Halfin-Whitt has been studied extensively in the recent literature. Atar was the first to study 
scheduling and routing control problem under infinite-horizon discounted cost. He has solved the 
scheduling control problem under a set of conditions on the network structure and parameters, and 
the running cost function (Assumptions 2 and 3 in [6]). Simplified models with either class only, 
or pool only dependent service rates under the infinite-horizon discounted cost are further studied 
in Atar et al. [7]. Gurvich and Whitt [EHm studied queue-and-idleness-ratio controls and their 
associated properties and staffing implications for multiclass multi-pool networks, by proving a 
state-space-collapse (SSC) property under certain conditions on the network structure and system 
parameters (Theorems 3.1 and 5.1 in [E]). Dai and Tezcan [T2l[T3] studied scheduling controls 
of multiclass multi-pool networks in the finite-time horizon, also by proving an SSC property 
under certain assumptions. Despite all these results that have helped us better understand the 
performance of a large class of multiclass multi-pool networks, there is a lack of good understanding 
of the behavior of the limiting controlled diffusions due to the implicit form of its drift. Our result 
on an explicit representation of the drift breaks this fundamental barrier. 

There is limited literature on ergodic control of multiclass multi-pool networks in the Halfin- 
Whitt regime. Ergodic control of the multiclass “V” model is recently studied in [T]. Armony [3] 
studied the inverted “V” model and showed that the fastest-server-first policy is asymptotically 
optimal for minimizing the steady-state expected queue length and waiting time. Armony and 
Ward [4] showed that for the inverted “V” model, a threshold policy is asymptotically optimal for 
minimizing the steady-state the expected queue length and waiting time subject to a “fairness” 
constraint on the workload division. Ward and Armony [26] studied blind fair routing policies 
for multiclass multi-pool networks, which is based on the number of customers waiting and the 
number of severs idling but not on the system parameters, and used simulations to validate the 
performance of the blind fair routing policies comparing them with non-blind policies derived from 
the limiting diffusion control problem. Biswas [8] recently studied a multiclass multi-pool network 
with “help” where each server pool has a dedicated stream of a customer class, and can help with 
other customer classes only when it has idle servers. In such a network, the control policies may 
not be work-conserving, and from the technical perspective, the associated controlled diffusion has 
a uniform stability property, which is not satisfied for general multiclass multi-pool networks. 

1.2. Organization. The rest of this section contains a summary of the notation used in the paper. 
In Section [2Tl we introduce the multiclass multi-pool parallel server network model, the asymptotic 
Halfin-Whitt regime, the state descriptors and the admissible scheduling and routing controls. 
In Section 12.21 we introduce the diffusion-scaled processes in the Halfin-Whitt regime and the 
associated control parameterization, and in Section [2.31 we state the limiting controlled diffusions. 
In Section 12.41 we describe the two formulations of the ergodic diffusion control problems. In 
Section [S] we first review the general model of controlled diffusions studied in [T] , and then state the 
general hypotheses and the associated stability results [Section l3.2p . We then study the associated 
ergodic control problems with constraints in Section 13.31 We focus on the recurrence properties 
of the controlled diffusions for multiclass multi-pool networks in Section SI The leaf elimination 
algorithm and the resulting drift representation are introduced in Section 14.11 and some examples 
applying the algorithm are given in Section 14.21 We verify the structural assumption of Section 13.21 
and study the positive recurrence properties of the limiting controlled diffusions in Section 14.31 We 
discuss some special cases in Section 14.41 The optimal stationary Markov controls for the limiting 
diffusions are characterized in Section [5l Some concluding remarks are given in Section [6l 

1.3. Notation. The following notation is used in this paper. The symbol M, denotes the field of 
real numbers, and M+ and N denote the sets of nonnegative real numbers and natural numbers, 
respectively. Given two real numbers a and b, the minimum (maximum) is denoted by a A 6 (a V 6), 
respectively. Define a"*" := a VO and a~ := —(a AO). The integer part of a real number a is denoted 
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by [aj. We use the notation Cj, i = 1,..., d, to denote the vector with entry equal to 1 and all 
other entries equal to 0. We also let e := (1,... , 1)'''. 

For a set ^4 C we use A, A'^, dA, and Ia to denote the closure, the complement, the boundary, 
and the indicator function of A, respectively. A ball of radius r > 0 in around a point x is 
denoted by Br{x), or simply as Bj- A x = 0. The Euclidean norm on is denoted by | • |, x • y, 
denotes the inner product of x,?/ G and ||3:|| := Yli=i\^i\- 

For a nonnegative function g G C(R'^) we let 0{g) denote the space of functions / G 
satisfying sup^-gj^d i+g{l) < This is a Banach space under the norm 


ll/ll. 


sup 


l/(^)l 

1 + gix) ■ 


We also let o(( 7 ) denote the subspace of 0{g) consisting of those functions / satisfying 


lim sup 

|x|—>-oo 


\f{x)\ 

1 + g{x) 


0 . 


Abusing the notation, 0{x) and o(x) occasionally denote generic members of these sets. For two 
nonnegative functions / and g, we use the notation f ^ g to indicate that f G 0{g) and g G 0(/). 

We denote by p > 1, the set of real-valued functions that are locally p-integrable and by 

functions in whose weak derivatives, i = 1,..., A:, are in 

The set of all bounded continuous functions is denoted by By we denote the set of 

functions that are fc-times continuously differentiable and whose derivatives are locally Holder 
continuous with exponent a. We define A: > 0, as the set of functions whose derivatives, 

i = 1,..., A:, are continuous and bounded in and denote by C^(R'^) the subset of (R'^) with 
compact support. For any path X{-) we use the notation AX(t) to denote the jump at time t. 

Given any Polish space X, we denote by V{X) the set of probability measures on X and we 
endow V{X) with the Prokhorov metric. Also B{X) denotes its Borel cr-algebra. By 6x we denote 
the Dirac mass at x. For v G V{X) and a Borel measurable map /: A —>■ R, we often use the 
abbreviated notation 

p(/) := [ fdu. 

Jx 

The quadratic variation of a square integrable martingale is denoted by (•, •) and the optional 
quadratic variation by [•,•]. For presentation purposes we use the time variable as the subscript 
for the diffusion processes. Also ai,A 2 ,... and C'i,C' 2 ,... are used as generic constants whose 
values might vary from place to place. 


2. Controlled Multiclass Multi-Pool Networks in the Halfin-Whitt Regime 

2.1. The multiclass multi-pool uetwork model. All stochastic variables introduced below are 
defined on a complete probability space P). The expectation w.r.t. P is denoted by E. We 

consider a sequence of network systems with the associated variables, parameters and processes 
indexed by n. 

Consider a multiclass multi-pool Markovian network with I classes of customers and J server 
pools. The classes are labeled as 1,..., / and the server pools as 1,..., J. Set I = {1,..., /} 
and JA = {1,...,J}. Customers of each class form their own queue and are served in the first- 
come-first-served (FCFS) service discipline. The buffers of all classes are assumed to have infinite 
capacity. Customers can abandon/renege while waiting in queue. Each class of customers can be 
served by a subset of server pools, and each server pool can serve a subset of customer classes. For 
each i G X, let J'{i) C J' be the subset of server pools that can serve class i customers, and for 
each j G J , let X(j) C X be the subset of customer classes that can be served by server pool j. For 
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each i € Z and j G J^ if customer class i can be served by server pool j, we denote i ^ j as an 
edge in the bipartite graph formed by the nodes in Z and J"; otherwise, we denote i oo j. Let £ be 
the collection of all these edges. Let G = {Z U J',£) be the bipartite graph formed by the nodes 
(vertices) ZVJ J and the edges £. We assume that the graph Q is connected. 

For each j G , let be the number of servers (statistically identical) in server pool j. 
Customers of class i £ Z arrive according to a Poisson process with rate A” > 0, i G X, and 
have class-dependent exponential abandonment rates 7 ” > 0. These customers are served at an 
exponential rate /rT > 0 at server pool j, if i ~ j, and otherwise, we set /i”- = 0. We assume that 
the customer arrival, service, and abandonment processes of all classes are mutually independent. 
The edge set £ can thus be written as 

X = {(i,j)eXx J : /rT>0}. 

A pair {i,j) G X is called an activity. 

2.1.1. The Halfin-Whitt regime. We study these multiclass multi-pool networks in the Halfin-Whitt 
regime (or the Quality-and-Efficiency-Driven (QED) regime), where the arrival rates of each class 
and the numbers of servers of each server pool grow large as n —)• 00 in such a manner that the 
system becomes critically loaded. In particular, the set of parameters is assumed to satisfy the 

hij hij > 0 ; 'J? ^ 7i > 0 

flij , ^/n (n“^AT - uj) 0 , 

where ^ij > 0 for i ^ j and ntj = 0 for i oo j. Note that we allow the abandonment rates to be 
zero for some, but not for all i G X. 

In addition, we assume that there exists a unique optimal solution (^*, p*) satisfying 

= p* = yj^J, (2.3) 

iex 

and > 0 for all i ^ j (all activities) in X, to the following linear program (LP): 

Minimize p 

subject to E 
i&J 

'y ^ ^ij — Pi j ^ 1 

iex 

^ij >0, i & Z, j ^ J. 

This assumption is referred to as the complete resource pooling condition Wz\. It implies that the 
graph ^ is a tree PEI]- Following the terminology in [SET], this assumption also implies that all 
activities in £ are basic since > 0 for each activity (i,j) or edge i ~ j in £. Note that in our 
setting all activities are basic. 

We define the vector x* = {x*)i^x and matrix z* = by 

< = 4 =^^/^- ( 2 - 4 ) 

j&J 

The vector x* = (x*) can be interpreted as the steady-state total number of customers in each 
class, and the matrix as the steady-state number of customers in each class receiving service, in 
the fluid scale. Note that the steady-state queue lengths are all zero in the fluid scale. The solution 


( 2 . 1 ) 

( 2 . 2 ) 


following: as n 

a: 


n 


00 , the following limits exist 
► Ai > 0, 

K - 


n 


Vj > 0. 


\/n 


1 pij) 
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to the LP is the steady-state proportion of customers in each class at each server pool. It is 
evident that (|2.3p and (|2.4p imply that 

e ■ X* = e ■ V, 


where v := {vj)j^j. 


2.1.2. The state descriptors. For each i £ I and j S J', let X” = {X^{t) : t > 0} be the total 
number of class i customers in the system, = {Q^{t) : f > 0} be the number of class i customers 
in the queue, Z”- = {Z^j{t) : t > 0} be the number of class i customers being served in server pool 
j, and YJ^ = {Yl^{t) : t > 0} be the number of idle servers in server pool j. Set X” = (X”')jgi, 
F” = Q"’ = and Z"' = The following fundamental equations hold: 

for each i S X and j £ J and f > 0, we have 

x”(«) = Q”(()+ zg((), 


X; = Ypit)+ Y, 

i6X(i) 


(2.5) 


xr(t)>o, Qrw>o, z/(f)>o, z-(t)>o. 


The processes X” can be represented via rate-1 Poisson processes: for each i £ X and f > 0, it 
holds that 




Xf{Q)+A^{X^t)- Y 



( 2 . 6 ) 


where the processes A”, 5”- and El} are all rate-1 Poisson processes and mutually independent, and 
independent of the initial quantities Xf’(O). 


2.1.3. Scheduling control. We only consider work conserving policies that are non-anticipative and 
preemptive. The scheduling decisions are two-fold: (i) when a server becomes free, if there are 
customers waiting in one or several buffers, it has to decide which customer to serve, and (ii) when 
a customer arrives, if she finds there are several free servers in one or multiple server pools, the 
manager has to decide which server pool to assign the customer to. These decisions determine the 
processes Z'^ at each time. 

Work conservation requires that whenever there are customers waiting in queues, if a server 
becomes free and can serve one of the customers, the server cannot idle and must decide which 
customer to serve and start service immediately. Namely, the processes Q"' and F” satisfy 

Q"(f)AF/(f) = 0 Vf~j, Vt>0. (2.7) 

Service preemption is allowed, that is, service of a customer can be interrupted at any time to 
serve some other customer of another class and resumed at a later time. Following [^, we also 
consider a stronger condition, joint work conservation (JWC), for preemptive scheduling policies. 
Specihcally, let XX be the set of all possible values of X”'(f) at each time t > 0 for which there is 
a rearrangement of customers such that there is no customer in queue or no idling server in the 
system and the processes and F” satisfy 

e • Q”(f) A e • F”(f) = 0, f>0. (2.8) 

Note that the set may not include all possible scenarios of the system state X"'(t) for finite n 
at each time f > 0. 

We define the action set U"(x) as 
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U"-(x) := \z e : Zij < Xi , Zij < N^, = Xi - ^ Zij , yj = ^ , 

% A yj = 0 Vz ~ j , e • g A e • y = o| . 

Then we can write Z'^{t) S for each t > 0. 

Define the tj-fields 

:= a{X^{0),A^{t),S:^it),Rnt) : iGl,jGj, 0<s<t}vM, 

and 

er := a{6Ant,r),6Srj{t,r),6Rntx) ■ r > O} , 

where M is the collection of all P-null sets, 

Ant) := Anxn), sAnt,r) ■.= Ant+r)-Ant), 

S^nt) ■■= S^J (y" Z^jis) ds) , SS^nt, r) := 5" (/r" Z^^^) ds + r) - S^j{t), 

and 

Rnt) := R7 [l? Qns) ds) , 5Rnt, r) := R^ ( 7 ? Q?i^) ds + Tfr) - i?r(t) ■ 

The filtration F” := : t > 0} represents the information available up to time t, and the 

filtration G” := {Q^ : t > 0} contains the information about future increments of the processes. 

We say that a scheduling policy is admissible if 

(i) the ‘balance’ equations in (I2.5ji hold. 

(ii) Z^{t) is adapted to 

(iii) Rf is independent of at each time t > 0; 

(iv) for each z S X and i ^ J, and for each t > 0, the process 6Sfj{t,-) agrees in law with 
S^jiAij •)) and the process 6RIt{t, ■) agrees in law with i?”( 7 P-). 

We denote the set of all admissible scheduling policies (Z”,F"', G”) by 3"'- Abusing the notation 
we sometimes denote this as Z^ £ 3”- 


2.2. Diffusion Scaling in the Halfin—Whitt regime. We define the diffusion-scaled processes 
X" = {Xf)i^x, Q" = , and Z" = , by 

1 


xnt) := ^{xnt)-nxn, 


n 


Qnt) := -^Qnt) 


n 


rm := -^rnt), 


(2.9) 


n 


By (12.41) . (12.5p . and (12.9p . we obtain the balance equations: for all t > 0, we have 

xnt ) = Q ?{ t )+ E 

j&Jii) 

Ynt)+ E = 0 


( 2 . 10 ) 


ARI ARAPOSTATHIS AND GUODONG PANG 


Also, the work conservation conditions in (|2.7I) . (I2.8p . translate to Qf{t) A = 0 for all i ^ j, 

and e • Q'^{t) A e • Y"'{t) = 0, respectively. By (I2.10p . we obtain 

= e-Q^{t)-e-Y^{t), 

and therefore the joint work conservation condition is equivalent to 

= (e-X”(t))+, e-Y^it) = (e-X"(t))". (2.11) 

In other words, in the diffusion scale and under joint work conservation, the total number of 
customers in queue and the total number of idle servers are equal to the positive and negative parts 
of the centered total number of customers in the system, respectively. 


Let 


’ vn 


: = 






R? 1? 


fQ?is)di 

Jo 


Q2{s)ds 


These are square integrable martingales w.r.t. the filtration F” with quadratic variations 

\n nV-. ft ft 

{Mim := := ^ / Z-(s)ds, := ^ / Qr(^)ds- 

n n Jq n Jo 

By (j2.6p . we can write as 


Ar(t) = xr(o)+o- 


Jo 


s-li / Qr('S)ds 




where is defined as 


C := 


+ , ( 2 . 12 ) 


^ V 




with z*j as defined in p2.4p . Note that under the assumptions on the parameters in p2.ip - 
the hrst constraint in the LP, it holds that 


e? 


A £i •— Aj ^ ^ 


"13 • 


and 
(2.13) 




Welet^:= (£i,..., 

2.2.1. Control parameterization. Define the following processes: for i G X, and t > 0, 


OV-ff) 

U^’^it) := { 

ei 


if e ■ Q^{t) >0 
otherwise. 


and for j G and t > 0, 


U^r^{t) := e.Y"(b 


ife-y'^(t)>0 


ej 


otherwise. 


(2.14) 


(2.15) 
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The process represents the proportion of the total queue length in the network at queue i 

at time t, while represents the proportion of the total idle servers in the network at station 

j at time t. Let ;= with ;= ,..., , and := ,..., . 

Given G 3” the process U” is uniquely determined via (I2.10p and (I2.14p - (I2.15I) and lives in the 
set 

U := {n = u^) gR^xR^ : e ■ = e ■ = l} . (2.16) 

It follows by (I2.10p and (I2.1ip that, under the JWC condition, we have that for each t > 0, 


Q^{t) = (e-X"(t))+17"’"(t), 

Y'^{t) = {e ■ X^{t)Y . 


(2.17) 


2.3. The limiting controlled diffusion. Before introducing the limiting diffusion, we define a 
mapping to be used for the drift representation as in mi- For any a G and /3 G M'^, let 

Dg := {(a,/?) G X : e • a = e • /3} , 

and define a linear map G : Dq —)■ such that 

Vi G X, 

j 

= /3j, Vj G J7, (2-18) 

i 

i/jij = 0 , yi oo j. 

It is shown in Proposition A.2 of [5] that, provided ^ is a tree, there exists a unique map G satisfying 
(j2.18p . We define the matrix 

^ := {'tpij)iGX,j&j = G{a,/3), for (q;,/3)gL>g- (2-19) 

Following the parameterization in Section [2.2.11 we define the action set U as in (j2.16p . We use 
and n® to represent the control variables for customer classes and server pools, respectively, 
throughout the paper. For each x gR^ and u = {u'^,u^) G U, define a mapping 

G[u]{x) := G{x — {e ■ xYu^j—Y ' xY'^^) ■ ( 2 . 20 ) 


Remark 2.1. The function G[u](a:) is clearly well defined for u = u^) = (0, 0), in which case we 

denote it by G^{x). See also Remark oi 


We quote the following result [H Lemma 3]. 

Lemma 2.1. There exists a constant cq > 0 such that, whenever G which is defined by 

X” := {x G Z:((_ : ||x — nx*|| < cqu} , (2-21) 

the following holds: If Q” G Z:(_ and G satisfy (e • QY A (e • YY = 0, then 

= G{X^ -YY 

satisfies Z^ G and (12.5p holds. 

Remark 2.2. It is clear from p2.5|) and (j2.10p that 

Z^{t) = G{X^{t)-Q^{t),N^-Y^{t)), 


Z^{t) = G{X^{t)-Q^{t),-Y^{t)) . 
Also, by (j2.17p . under the JWC condition, we have 

Z^{t) = G[U^{t)]{X^{t)). 






















10 


ARI ARAPOSTATHIS AND GUODONG PANG 


Note that the requirement that (X” — — Y^) G Dq is an implicit assumption in the 

statement of the lemma. As a consequence of the lemma, "XY C X". Thus, asymptotically as 
n —)• oo, the JWC condition can be met for all diffusion scaled system states. 

Definition 2.1. We say that Z” G 3”' is jointly work conserving (JWC) in a domain D C if 
(12.81) holds whenever X^{t) G D. We say that a sequence {Z"" G 3”, n G N} is eventually jointly 
work conserving (EJWC) if there is an increasing sequence of domains C n G N, which 
cover and such that each Z” is JWC on We denote the class of all these sequences by 3- 
By Lemma l2.1l the class 3 is nonempty. 


Under the EJWC condition, the convergence in distribution of the diffusion-scaled processes 
to the limiting diffusion X in (|2.22p can be proved [6l Proposition 3]. 

The limit process X is an /-dimensional diffusion process, satisfying the ltd equation 


dXt = b{Xt,Ut)dt + YdWt, ( 2 . 22 ) 

with initial condition Xq = x and the control Ut G U, where the drift 6 : x U ^ takes the 

form 

bi{x,u) = bi{x,{u‘',u'')) ^ HijGij[u]{x) --y^e ■ x)~^u'i + ii VfGX, (2.23) 

and the covariance matrix is given by 

E := diag(v^^,..., v^^) . 

Let 11 be the set of all admissible controls for the limiting diffusion (see Section EH). 

The limiting processes Q, Y, and Z satisfy the following: Qj > 0 for i G X, > 0 for j G J, 
and for all t > 0, and it holds that 


Xi{t) = Qi{t)+ Y, Zijit) 


(2.24) 


Y,{t)+ Y = 0 VjGj. 

iGXL) 

Note that these ‘balance’ conditions imply that joint work conservation always holds at the diffusion 
limit, i.e., 

e-Q{t) = {e-X{tY, e-Y(t) = {e-X{t))~ Vt>0. (2.25) 

It is clear then that by (I2.18P and (I2.25p . we have 

Z{t) = G{X{t)-Q{t),-Yit)). 


2.4. The limiting diffusion ergodic control problems. We now introduce two formulations 
of ergodic control problems for the limiting diffusion. 

(1) Unconstrained control problem. Dehne the running cost function r : x U —>■ by 

r{x,u) = r(3:, (n^ n®)), 


where 

/ J 

r{x,n) = [(e-x)+rj;e*«)™ + [(e-a;)-rj;0(«*)™, m>l, (2.26) 

i=l j=l 

for some positive vectors C = (Cij ■ ■ ■ > C/)"'" a-iid C = (Ci; • • • > Cj)"*"- 

The ergodic criterion associated with the controlled diffusion X and the running cost r is dehned 
as 


1 


u 


Jx,u[r] ■■= limsup - E), 


T—^oo 


r{Xt,Ut)dt 


t/Git. 


(2.27) 
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The ergodic cost minimization problem is then defined as 

Q*{x) = M^Jx,u[r]- (2.28) 

The quantity g*{x) is called the optimal value of the ergodic control problem for the controlled 
diffusion process X with initial state x. 

(2) Constrained control problem. The second formulation of the ergodic control problem is as 
follows. The running cost function ro(x,u) is as defined in (|2.26l) with (" = 0. Also define 

rj{x,u) := [(e-x)"u*]™, (2.29) 

and let 5 = (5i,..., 5j) be a positive vector. The ergodic cost minimization problem under idleness 
constraints is defined as 


Q*c{x) = Jx.uVo] 


subject to Jx,u[T'i\ < ) 3 ^ J ■ 

The constraint in (j2.,Sljl can be written as 


lim — WE 
T^oo T 


r(-s 


G,j[U]{Xt)] dt 


iGXO') 




j^J- 


(2.30) 

(2.31) 


As we show in Section [3l the optimal values Q*{x) and ^*(a;) do not depend on x G and thus 
we remove their dependence on x in the statements below. We prove the well-posedness of these 
ergodic diffusion control problems, and characterize their optimal solutions in Sections 0] and [5j 


3. Ergodic Control of a Broad Class of Controlled Diffusions 

We review the model and the structural properties of a broad class of controlled diffusions for 
which the ergodic control problem is well posed [1]. We augment the results in [T] with the study 
of ergodic control under constraints. 

3.1. The model. Consider a controlled diffusion process X = {Xt, t > 0} taking values in the 
d-dimensional Euclidean space and governed by the Ito stochastic differential equation 

dw = b{Xt,Ut)dt + a{Xt)dWt. (3.1) 

All random processes in (|3.ip live in a complete probability space (D,5^, P). The process W is a d- 
dimensional standard Wiener process independent of the initial condition Xq. The control process 
U takes values in a compact, metrizable set U, and Ut{uj) is jointly measurable in (t, uj) G [0, oo) x D. 
Moreover, it is non-anticipative: for s < t, Wt — Wg is independent of 

■= the completion of (t{Xq, Ur, Wr, r < s} relative to (3^, P). 

Such a process U is called an admissible control. Let it denote the set of all admissible controls. 

We impose the following standard assumptions on the drift b and the diffusion matrix ct to 
guarantee existence and uniqueness of solutions to equation (13.ip . 

(Al) Local Lipschitz continuity: The functions 

b = [bi, ..., 6^]"^ : X U ^ , and a = [ 0 ^^] : ^ 

are locally Lipschitz in x with a Lipschitz constant Cr > 0 depending on i? > 0. In other 
words, for all x, y G Br and u G U, 

\b{x,u)-b{y,u)\ + \\a{x)-(y{y)\\ < CR\x-y\. 

We also assume that b is continuous in {x,u). 
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(A2) Affine growth condition: h and cr satisfy a global growth condition of the form 
|6(a:,+ ||a(x)|p < Ci(l + |xl^) V(x,u) € x U, 
where HolP := trace (oa"'') . 

(A3) Local nondegeneracy: For each ii > 0, it holds that 


d 

yxeBji, 

*j=i 

for all = (.^ 1 ,..., G where o := ao'*'. 

In integral form, m is written as 

b{X„Us)ds+ [\{X,)dW,. (3.2) 

Jo 

The third term on the right hand side of (j3.2p is an Ito stochastic integral. We say that a process 
X = {Af(n;)} is a solution of (13.11) . if it is ^i-adapted, continuous in t, defined for all w G and 
t G [0,oo), and satisfies (13.21) for all t G [0,oo) a.s. It is well known that under (Al)"(A3), for any 
admissible control there exists a unique solution of (j3.ip [21 Theorem 2.2.4]. 

The controlled extended generator £“ of the diffusion is defined by C^(M'^) —)• C(M'^), where 
li G U plays the role of a parameter, by 



d 

■= ^ aij{x)dijf{x)+ '^bi{x,u)dif{x), uGU. 




(3.3) 


i=l 


a 

dxi 




and dij 


We adopt the notation di := 

Of fundamental importance in the study of functionals of X is Ito’s formula. For f £ 
and with as defined in (13.3p , it holds that 


f{Xt) = f{Xo) + f C^^fiXs) ds + Mt, 
Jo 


a.s., 


(3.4) 


where 


AT := 


[\vf{Xs),(y{Xs)dWs) 

Jo 


is a local martingale. Krylov’s extension of Ito’s formula m p. 122] extends (13.41) to functions / 
in the local Sobolev space W^^(](M'’*), p> d. 

Recall that a control is called Markov if Ut = v{t,Xt) for a measurable map v: M+ x R'’* U, 
and it is called stationary Markov if v does not depend on t, i.e., n: R'’* ^ U. Correspondingly 
(13.11) is said to have a strong solution if given a Wiener process on a complete probability 

space (fl,3’,R), there exists a process X on (0,5, R), with Xq = xq G R'’*, which is continuous, 
S’radapted, and satisfies (13.2p for all t a.s. A strong solution is called unique, if any two such 
solutions X and X' agree R-a.s., when viewed as elements of C([0,oo),R'’*). It is well known that 
under Assumptions (Al)-(A3), for any Markov control v, (j3.ip has a unique strong solution |18j . 

Let ilsM denote the set of stationary Markov controls. Under v G ilsM, the process X is strong 
Markov, and we denote its transition function by Py{x, •). It also follows from the work of mm 
that under v G ilsM; the transition probabilities of X have densities which are locally Holder 
continuous. Thus defined by 


hi=i 


- d 

C'^fix) := -^aij{x)dijf{x) + ^bi[x,v{x))dif{x), n G ilsM 

i=l 
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for / G is the generator of a strongly-continuous semigroup on which is strong 

Feller. We let P" denote the probability measure and the expectation operator on the canonical 
space of the process under the control v G ilsM, conditioned on the process X starting from x G 
at t = 0. 

Recall that control v G ilsM is called stable if the associated diffusion is positive recurrent. We 
denote the set of such controls by ilsSM, and let denote the unique invariant probability measure 
on for the diffusion under the control v G ilsSM- We also let := : n G Hssm}) and S 

denote the set of ergodic occupation measures corresponding to controls in ilsSM; that is, 

g := |7tGP(M'^ xU) : [ C^f{x)n{dx,du) = 0 V/ G j , 

I JR'^xU J 

where f{x) is given by (j3.3p . 

We need the following definition: 

Definition 3.1. A function /i: x U ^ M is called inf-compact on a set A C if the set 

An {x : min^gu h{x, u) < c} is compact (or empty) in for all c G M. When this property holds 
for A = then we simply say that h is inf-compact. 


Recall that v G ifsSM if and only if there exists an inf-compact function V G a bounded 

domain D and a constant e > 0 satisfying 

CV{x) < -e VxGD^ 

We denote by t(A) the first exit time of a process {Xt, t G K-(-} from a set A C defined by 

t(A) := inf {t > 0 : Xt ^ A} . 

The open ball of radius R in centered at the origin, is denoted by Bji, and we let Xji := 
and Xr := x{Bfi). 

We assume that the running cost function r(x, u) is nonnegative, continuous and locally Lipschitz 
in its first argument uniformly in n G U. Without loss of generality we let Cr be a Lipschitz constant 
of r( •, u) over Br. In summary, we assume that 

(A4) r: X U —M+ is continuous and satisfies, for some constant Cr > 0 

|r(x,u) - r(y,u)| < Cr\x - y\ \/x,y e Br , \/u G V, 
and all R > 0. 

In general, U may not be a convex set. It is therefore often useful to enlarge the control set to 
R(U). For any v{du) G R(U) we can redefine the drift and the running cost as 


b{x,v) := / b{x,u)v{du), and r{x,v) := / r{x,u)v{du). (3.5) 

Jv Jv 

It is easy to see that the drift and running cost defined in (13.51) satisfy all the aforementioned 
conditions (A1)-(A4). In what follows we assume that all the controls take values in R(U). These 
controls are generally referred to as relaxed controls, while a control taking values in U is called 
precise. We endow the set of relaxed stationary Markov controls with the following topology: 
—)• u in iIsM if and only if 



g{x,u)vn{du 


x) dx 


- 

n—!*-oo 



g{x, u)v{du 


x) dx 


for all / G L^(M'^) D L^(M'^) and g G x U). Then Hsm is a compact metric space under this 

topology O Section 2.4]. We refer to this topology as the topology of Markov controls. A control 
is said to be precise if it takes value in U. It is easy to see that any precise control Ut can also be 
understood as a relaxed control by Ufidu) = Abusing the notation we denote the drift and 
running cost by b and r, respectively, and the action of a relaxed control on them is understood as 
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in (13.5p . In this manner, the definition of Jx,u[^] in (12.27p . is naturally extended to relaxed C/ G it 
and X G For v G ilssM, the functional Jx,v[r] does not depend on x G In this case we drop 
the dependence on x and denote this by Jv[r]. Note that if 7T^(dx,dtt) := Hx{dx) v{du \ x) is the 
ergodic occupation measure corresponding to u G ilssMj then we have 


Jv[r] 


/ r{x,u) ny{dx,du). 
jR'ixV 


Therefore, the restriction of the ergodic control problem in (I2.28p to stable stationary Markov 
controls is equivalent to minimizing 



r{x, u) 7t(dx, du) 


over all 7t G S- If the infimum is attained in S, then we say that the ergodic control problem is 
well posed, and we refer to any ft G 9 that attains this infimum as an optimal ergodic occupation 
measure. 


3.2. Hypotheses and review of some results from [T]. A structural hypothesis was introduced 
in [T] to study ergodic control for a broad class of controlled diffusion models. This is as follows: 

Hypothesis 3 . 1 . For some open set K, C M'^, the following hold: 

(i) The running cost r is inf-compact on JC. 

(ii) There exist inf-compact functions V G and h G x U), such that 

£“V(x) < l — h{x,u) V (x, ti) G X U , 

£“V(x) < I-|-r(x,tt) V (x, u) G/C X U . 

Without loss of generality, we assume that V and h are nonnegative. 

In Hypothesis 13.11 for notational economy, and without loss of generality, we refrain from using 
any constants. Observe that for /C = the problem reduces to an ergodic control problem with 
inf-compact cost, and for fC = 0 we obtain an ergodic control problem for a uniformly stable 
controlled diffusion. As shown in [T], Hypothesis 13.II implies that 

Jx,u[hlic '^xu] < Jx,u[rlicxu] VI/Gil. 

The hypothesis that follows is necessary for the value of the ergodic control problem to be finite. 
It is a standard assumption in ergodic control. 

Hypothesis 3 . 2 . There exists 1/ G il such that jj[r] < oo for some x G 

It is shown in [T] that under Hypotheses 13.II and 13.21 the ergodic control problem in (j2.27p - (l2.28p 
is well posed. The following result which is contained in Lemma 3.3 and Theorem 3.1 of m plays 
a key role in the analysis of the problem. Let 

T-L := (/C X U) U {(x, u) G X U : r{x,u) > /i(x,ri)} , 

where K, is the open set in Hypothesis 13.11 

Lemma 3 . 1 . Under Hypothesis \3.1\ the following are true. 

(a) There exists an inf-compact function h G C{W^ x U) which is locally Lipschitz in its first 
argument uniformly w.r.t. its second argument, and satisfies 

ko 

r{x,u) < h{x,u) < — (l-i/i(x, tt) I^c(x, u)-i r(x, u) I^(x, u)) 
for all (x,u) G X U, and for some positive constant ko > 2. 


(3.6) 
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(b) The function V in Hypothesis \3.1\ satisfies 

£“V(x) < I — h{x,u)I'uc(x,u) + r{x,u)I-u{x,u) V(x,it) G x U. 

(c) It holds that 

Jx,um < ko{l + Jx,u[r]) Vi/Gil. (3.7) 

Hypothesis 13.21 together with (j3.7p imply that J^(j[h] < oo. This together with the fact that 

h is inf-compact and dominates r is used in [T] to prove that the ergodic control problem is well 
posed. Also, there exists a constant g* such that 


= inf limsup — E(: 
ueli T^oo T 


rriXt,Ut)dt 

Jo 


Vx G 


(3.8) 


Moreover, the infimum in (13.81) is attained at a precise stationary Markov control, and the set of 
optimal stationary Markov controls is characterized via a HJB equation that has a unique solution 
in a certain class of functions [H Theorems 3.4 and 3.5]. 

Another important result in [T] is an approximation technique which plays a crucial role in the 
proof of asymptotic optimality (as n —)• oo) of the Markov control obtained from the HJB for the 
ergodic control problem of the multiclass single-pool queueing systems. In summary this can be 
described as follows. We truncate the data of the problem by fixing the control outside a ball in M'^. 
The control is chosen in a manner that the set of ergodic occupation measures for the truncated 
problem is compact. We have shown that as the radius of the ball tends to infinity, the optimal 
value of the truncated problem converges to the optimal value of the original problem. 

The precise definition of the ‘truncated’ model is as follows. 


Definition 3.2. Let vq G ilssM be any control such that nvoi^) < oo- We fix the control xq on the 
complement of the ball Bn and leave the parameter u free inside. In other words, for each i? G N 
we define 


\b{x,vo{x)) 

if (x, u) G Bn X U, 
otherwise. 

(3.9) 

R( N jrix,u) 

r {x,u) := < 

(^r(x,Xo(x)) 

if (x, u) G Bn X U, 

otherwise. 

(3.10) 


Consider the ergodic control problem for the family of controlled diffusions, parameterized by 
7? G N, given by 

dW = b^{Xt,Ut)dt + (y{Xt)dWt, (3.11) 

with associated running costs r^{x,u). We denote by itsM(77, xq) the subset of Hsm consisting of 
those controls v which agree with uq on B'fi, and by 9{R) we denote the set of ergodic occupation 
measures of (I3.11j) . 

Let rjo := Tty^(h). By (13.7h . ryo is finite. Let (po G for any p > d, he the minimal 

nonnegative solution to the Poisson equation (see O Lemma 3.7.8 (ii)]) 

C^°ipo{x) = r]Q — h{x,vo{x)) x G M'’*. (3-12) 

Under HvDotheses l3.ll and l3.2l all the conclusions of Theorems 4.1 and 4.2 in [1] hold. Consequently, 
we have the following lemma. 

Lemma 3.2. Under Hvvotheses \3.1\ and \3.i^ the following hold. 

(i) The set 9{R) is compact for each R > 0, and thus the set of optimal ergodic occupation 
measures for r^ in 9{R), denoted as 9(7?), is nonempty. 

(ii) The collection Uij>o 9(7?) is tight in 7^(M'^ x U). 
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Moreover, provided tpQ € 0(min„gu h{- ,u)), for any collection {n^ S 9{R) ■ R > 0}, we have 

(iii) Any limit point of as R ^ oo is an optimal ergodic occupation measure of dsn for r. 

(iv) It holds that limj:j^oo Ti^{r^) = Q* ■ 


3.3. Ergodic control under constraints. Let — )• M_|_, 0 < i < A:, be a set of continuous 

functions, each satisfying (A4). Define 

k 

r := (3.13) 

1=0 

We are also given a set of positive constants 6j, i = 1,... ,k. The objective is to minimize 

= / ro(x, tt) 7t(dx, drt) (3.14) 


over all 7T G S, subject to 


7T(rj) = / ri{x,u) n{dx,du) < 8i, i = l,...,k. 
iK'^xU 

For 6 = (5i,... , 5^) G let 

d{(5) := {tt G g : 7T(ri) < 6*, i = l,...,k}, 

d{°(6) := {tt G g : 7t(ri) < 5*, i = l,...,k} . 


(3.15) 


(3.16) 


It is straightforward to show that 34(5) is convex and closed in g. Let 3{e(6) (ge) denote the set of 
extreme points of 3{(5) (g). 

Throughout this section we assume that Hypothesis 13.11 holds for r in (I3.13P without any further 
mention. We have the following lemma. 


Lemma 3.3. Suppose that 

34(6) n {tt G g : TT(ro) < oo} / 0 . 

Then there exists n* G 34(6) such that 

7t*(ro) = inf 7T(ro). 

‘K{&) 

Moreover, n* may he selected so as to correspond to a precise stationary Markov control. 


Proof. By hypothesis, there exists 6o G M+ such that 34 := 34(6) n (tt G g : TT(ro) < 6o} ^ 0. By 
(13.71) we have 

k 

n{h) < ko + ko + kon{ro) V7TG3{(6), (3.17) 

i=l 

which implies, since h is inf-compact, that 34 is pre-compact in x U). Let 7t„ be any sequence 
in 34 such that 


7tn(i’o) -S' £*0 

n—>-oo 


inf 

7ieM(5) 


7T(ro) . 


By compactness tt^ —)■ tt* G ^(M'^ x U) along some subsequence. Since g is closed in P(M'^ x U), it 
follows that Tt* G g. On the other hand, since the functions r* are continuous and bounded below, 
it follows that the map tt i—?■ TT(ri) is lower-semicontinuous, which implies that TT*(ro) < go ^-iid 
< Si for i = 1,... ,k. It follows that tt* G 34 C 34(6). Therefore, 34 is closed, and therefore 
also compact. 

Applying Choquet’s theorem as in the proof of [21 Lemma 4.2.3], it follows that there exists 
ft* G 34e, the set of extreme points of 34, such that fi*(ro) = go- On the other hand, we have 
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C Se by [a Lemma 4.2.5]. It follows that tt* S fK(6)nSe- Since every element of Se corresponds 
to a precise stationary Markov control, the proof is complete. □ 

Definition 3.3. We say that the vector 6 G (0, oo)^ is feasible (or that the constraints in (13.151) 
are feasible) if there exists tL G 1K°(6) such that 7t'(ro) < oo. 

Lemma 3.4. //6 G (0, oo)^ is feasible, then 5 i—infj,.g7T(ro) is continuous at 8. 

Proof. This follows directly from the fact that, since 5 is feasible, the primal functional 

6 i-A inf {7t(ro) : 7T(ri) < 8i ,i = 1,... ,k} 

7tGS 

is bounded and convex in some ball centered at 6 in M^. □ 

Definition 3.4. For 6 G and A = (Ai ..., A^)"'' G define the running cost by 

k 

gb,\{x,u) := rQ{x,u)+ '^\{ri{x,u) - hi) . 

i=\ 

Also, for /3 > 0 and 6 G (0, oo)^, we define the set of Markov controls 

Up{8) := {u G UssM : 7t„ G fK(5), 7t„(ro) < 13} , 
and let 34^(6) denote the corresponding set of ergodic occupation measures. 

Lagrange multiplier theory provides us with the following. 

Lemma 3.5. Suppose that 8 is feasible. Then the following hold. 

(i) There exists A* G sueh that 

inf 7t(ro) = inf n{gs,^*). (3.18) 

7r€M(6) TteS 

(ii) Moreover, for any n* G 34(6) that attains the infimum o/tt i—)• 7T(ro) in 33(8), we have 

n*(ro) = n*(gs,x*), 

and 

7t*(g5,A) < 7T*(ff6,A*) < n(gs,X*) V (tT, A) G S X . 

Proof. The proof is standard. See [20l pp. 216-221]. □ 

We next state the associated dynamic programming formulation of the ergodic control problem 
under constraints. Recall that Tg denotes the first hitting time of the ball B^, for e > 0. 

Theorem 3.1. Suppose that 8 G (0,oo)^ is feasible. Let A* G be as in Lemma WfR and n* be 
any element of 33(8) that attains the infimum in (I3.18p . Then, the following hold. 

(a) There exists a ^ C^(M'^) satisfying 

va.m.[C'^Lp,,(x) + gi^x*(x,u)] =n*(gs^x*), x G . (3.19) 

ueu 

(b) With V as in Hypothesis \S. 11 we have 99 * G 0(V), and ipf G o(V). 

(c) A stationary Markov control v G ilsSM is optimal if and only if it satisfies 

min 3ls,x*(x,V(p^(x);u) = b(x,v(x)) ■V(p^(x) + gs,x*{x,v(x)) , x G , (3.20) 

where 

31b,\-(x,p]u) := 6(x,u) •p + 55,a*(x,u) . 
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(d) The function has the stochastie representation 


ip Ax) = lim inf 

£\o ^;GU/3>0 ^/3(S) 


= lim E^ 
e\0 


J (^9S^*{Xs,v{Xs)) -7T*(56^a*)) ds 

(^gs^*{Xs,v{As)) ds 


for any v G Hsm that satisfies (|3.20l) . 

Proof. Let v* G ilssM satisfy 7 T*(dx,dn) := fiy* {dx) v*{du \ x). Since Tp{g^^x*) < oo, there exists 
a function ip^ G for any p > d, and such that (/J*(0) = 0, which solves the Poisson 

equation [21 Lemma 3.7.8 (ii)] 

C'^ip^ix) + gi,^^*{x,v*{x)) =n*{gs,\*), x G , 


(3.21) 


and satisfies, for all e > 0 , 


(p*{x) = E' 


gS,7,*{Xs,V*{Xs)) - 7T*(sr6^A*)) ds + (^*(X^J 


Vx G 


Let ii > 0 be arbitrary, and select a Markov control vn satisfying 

Argmin^gU IRa*( x,V(^*(x);n) if |x| < R, 
v*(x) otherwise. 


vr(x) = 


It is clear that vji G ilssM, and that if 7 t/j denotes the corresponding ergodic occupation measure, 
then we have 7 tR(r) < oo. It follows by (I3.2ip and the definition of vji that 

+56,A*(x,Xij(x)) < 7T*(g'6,A*)! X G . (3.22) 

By (|3.22p using [21 Corollary 3.7.3] we obtain 

7Tr(S'6,A*) < 7T*(S'6,A*)- 

However, since 7 tj:j(g '5 a*) > 7t*(g5,^*) by Lemma [331 it follows that we must have equality in (I3.22p 
a.e. in M'^. Therefore, since R > 0 was arbitrary, we obtain ()3.19l) . By elliptic regularity, we have 
ip^: G This proves part (a). 

Continuing, note that by (|3.17l) we have 7 T*(/i) < oo, and moreover that sup 7 jgj£^( 5 ) 7 t(/i) < oo 
for all /3 > 0. Thus we can follow the approach in Section 3.5 of [T], by considering the perturbed 
problem with running cost of the form 55 a* + sh and then take limits as e \ 0. Parts (b)~(d) then 
follow as in Theorem 3.4 and Lemma 3.10 of pQ. □ 


Concerning uniqueness, the analogue of Theorem 3.5 in [T] holds, which we quote next. The 
proof follows that of [H Theorem 3.5] and is therefore omitted. 

Theorem 3.2. Let the hypotheses of Theorem \3.1\ hold, and G C^PR*^) x R &e a solution of 

mill [£“(,5(x)+ 56 ,A*(x,tt)] = g, (3.23) 

such that (p~ G o(V) and <^(0) = 0. Then the following hold: 

(a) Any measurable selector v from the minimizer of ()3.23p is in ilssM and 7X.y{gi,^\*) < 00 . 

(b) If either g < 7X*{gs^},*), or ip ^ 0(min„gu h{-,u)'), then necessarily g = Tf{gi,^\*), and 

(p = tp^. 
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We finish this section by presenting an analogues to [H Theorems 4.1 and 4.2] (see also Lemma r3.2l) 
for the ergodic control problem under constraints. Let vq G Ussm be any control such that 
7ryg(r) < oo. For J = 0,1,..., k, define the truncated running costs relative to rj as in (13.101) . We 
consider the ergodic control problem under constraints in (|3.14j) - (l3.15jl for the family of controlled 
diffusions, parameterized by i? G N, given by (13.111) with running costs = r^(x, u), j = 0,1,... , fc. 
Recall that, as defined in SectionS(R) denotes the set of ergodic occupation measures of (|3.1ip . 
We also let 1K(5; R), 1K°(6; R) be defined as in (|3.16p relative to the set S(i?). We have the following 
theorem. 

Theorem 3.3. Suppose that 8 G (0, oo)^ is feasible, and that ipQ defined in (I3.12p satisfies ipo G 
©(min^jeu h{-,u)). Then the following are true. 

(a) There exists Rq > 0 such that 

2{°(6;R)n{7TG 9{R) : 7t(ro) < oo} / 0 VR>Ro- 

(b) It holds that 

inf 7r(ro) -)• inf 7r(ro). 

7reM(5;R) R-^oo 


Proof Let e > 0 be given. By Lemma 13.41 for all sufficiently small e > 0, there exist 6f < 6j, 
i = 1,... ,k, such that 6^ is feasible and 

inf 7r(ro) < inf 7T(ro) + |. (3.24) 

"651(5^) 7tG3f(S) 

For e > 0, let := tq + e h. By (13.71) we have 

k 

7t(ro) < n{rfij < {I + kQ£)n{rQ) + koi + V7rel{(5). 

i=l 

Therefore, for any e > 0, we can choose e > 0 small enough so that 


inf 7T(^e) < inf 7 t(?’o) + 

7IgM(6£) 7IgM(6£) 


£ 

4 • 


(3.25) 


:= rir(x,u) + J^Ai(ri(x,u) - 5f) . 

Z=1 

By Lemmas 13.31 and 13.51 there exist A* G and n* G 1K(6^) such that 

n*(ri) = inf n(ri) = inf 7t(gg^se^p,*) = n* . 

'n:gJ{(6'^) 7tg!K(5®) 


(3.26) 


Define the truncated running cost gf^ex* relative to ge, 5 '^,x* as in (I3.inp . Since x*) 

is finite, the hypotheses of Lemma [3.21 are satisfied. Let S(7?) denote the collection of ergodic 
occupation measures in S(i?) which are optimal for for gf^e x*- Therefore, it follows by Lemma 13.21 
that {S(7?) : i? > 0} is tight, and any limit point of 7t^ G 9{R) as R —>■ oo satisfies (I3.26p . Since 
ri < h it follows by dominated convergence that 

limsup 'n"^(rf^) < 6f < 5*, i = l,...,k. 

i?— >oo 

which establishes part (a). 

Therefore, there exists Rq > 0 such that tt^ G 34(6,7?) for all R> Rq, and by ()3.26p . 

< inf n{rg) + | \/ R > Rq . 

7tg5{(6®) ^ 


(3.27) 
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Combining (|3.24p - (l3.25p and (I3.27P we obtain 

< inf 7t(ro) + e, 

7ieM(S) 

which establishes part (b). The proof is complete. □ 


Let 5 S (0,oo)^ and i? > 0. Provided 7 ^ 0 we denote by ■ ■ ■ ,A^ jij)"'' G 

any such vector satisfying 


inf Ttfro) 
7tG5^(6,R) 


inf 

7tGS(R) 


7^(56,A|j) , 


and by 7 t|j, any member of lK(5,ii) that attains this infimum. It follows by Theorem 13.31 fal that, 
provided 1K°(5) 7 ^ 0 , then 1K°(6; R) ^ 0 for all R sufficiently large. Clearly, 7 t|j satisfies (13.1711 and 
R I—>• 7 Tjj(ro) is nonincreasing. Therefore { 7 t|j} is a tight family. It then follows by Theorem 13.31 (b) 
that any limit point of as 7? —>■ 00 attains the minimum of tt —)• 7T(ro) in 1K(6). Concerning the 
convergence of the solutions to the associated HJB equations we have the following. 


Theorem 3.4. Suppose that 8 G (0,oo)^ is feasible. Let denote the controlled extended generator 
corresponding to the diffusion in (13.1111 . ipo be as in (I3.12h . andXf^, g^^, defined as in (j3.10p relative 
to the running cost gs^\*, 7 t|j be as defined in the previous paragraph. Then there exists Rq > 0 such 
that for all R > Rq the HJB equation 

min + (x,u)] = 7t|j(ro), (3.28) 

has a solution Vr in for any p > d, with Vr{0) = 0, and such that the restriction of Vr 

on Br is in C‘^{Br). Also, the following hold: 

(i) there exists a constant Cq, independent of R, such that Vr < Cq + 2ipQ for all R > Rq; 

(ii) {Vr)~ G o(V + ipo) uniformly over R > Rq; 

(iii) Every nfi corresponds to a stationary Markov control v G ilssM that satisfies 

imn[b^{x,u)-WR{x) + g^^^,^{x,u)] = b{x,v{x))-VVRix) + gsXiiix,vix)), a.e.xGM'^. (3.29) 

Let and A* be as in Theorem \3.1\ . Then, under the additional hypothesis that 

po G 0(min h{- , u)) , 

for every sequence R 00 there exists a subsequence along which it holds that Vr —>■ and 

A|j ^ A*. Also, if Vr is a measurable selector from the minimizer of (I3.29P then any limit point 
of VR in the topology of Markov controls as 7? —)• 00 is a measurable selector from the minimizer of 


Proof. We can start from the perturbed problem with running cost of the form gs x* +eh to establish 
(j3.29p in Section 3.5 of [T], and then take limits as e 0. Parts (i) and (ii) can be established by 
following the proof of [H Theorem 4.1]. Convergence to (I3.20p as 7? 00 follows as in the proof 

of [U Theorem 4.2]. □ 


4. Recurrence Properties of the Controlled Diffusions 

In this section, we show that the limiting diffusions for a multiclass multi-pool network satisfy 
Hypothesis 13.II relative to the running cost in (12.2611 for any value of the parameters. Also, provided 
7 7 ^ 0, Hypothesis 13.21 is also satisfied. The proofs rely on a recursive leaf elimination algorithm 
which we introduce next. 
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4.1. A leaf elimination algorithm and drift representation. We now present a leaf elimi¬ 
nation algorithm and prove some properties. Recall the linear map G defined in (12.181) and the 
associated matrix 'h in (j2.19ll . and also the map G defined in (12.201) . 

Definition 4.1. Let Q{lU J',£,{a,l3)) denote the labeled graph, whose nodes are labeled by 
{a,f3), i.e., each node i has the label ai, and each node j ^ J has the label /3j. The graph Q 
is a tree and there is a one to one correspondence between this graph and the matrix T = /?) 

dehned in (j2.19p . We denote this correspondence by T ~ 

Let denote the (/ — 1) x J submatrix of T obtained after eliminating the row of iL. 

Similarly, is the 1 x (J — 1) submatrix resulting after the elimination of the column. 

If ? G X is a leaf of ^ (X U J", X, (a, /?)) , we let jj G J denote the unique node such that (z, ji) G £ 
and define 

(o,/3)^ 1 ? G^2-1-1 ? • • • ? •> Pi ; ■ ■ ■ ; — Pji + 1‘ ‘ ‘ ? Pj ) ? 

i.e., (a, /3)(-0 G is the vector of parameters obtained after removing a* and replacing Pj. 

with Pj. — aj. Similarly, if j G ^7 is a leaf, we define ij and {a,P)(^_j-^ in a completely analogous 
manner. □ 

Lemma 4.1. If i G I and/or j £ J' are leafs of G(I U J' ,£, {a, P)), then 

T(_j)(a,/3) ~ g(lU {J\{j}),£\{{ij,j)},{a,P)^_j)y 

Proof. If z G X is a leaf of ^(X U J, £, (a, /3)), then is the unique non-zero element in the 
row of 'I'(a:, P). Therefore, the equivalence follows by the fact that the concatenation of 'I'^“*)(q:, P) 
and row i of 'I'(q;, P) has the same row and column sums as 'I'(a, P). Similarly if j G is a leaf. □ 

Definition 4.2. In the interest of simplifying the notation, for a labeled tree g = ^(XU^X, £, (a, P)) 
we denote 

:= g{{X\{i})VJj,£\{{t,ji)]/a,P)^-y, 

and 

g(_j) := g(xu(J\{i}),X\{(ij,i)},(a,/3)(_j)), 

for leaves ? G X and 3 G J, respectively. 

We now present a leaf elimination algorithm, which starts from a server leaf elimination. A 
similar algorithm can start from a customer leaf elimination. 

Leaf Elimination Algorithm. Consider the tree g = g(XVJ J,£, (a,/?)) as described above. 
Server Leaf Elimination. Let ^Xieaf C ^X be the collection of all leaves of g which are members of J . 
We eliminate each j G jTieaf sequentially in any order, each time replacing g by ^(-j) and setting 
fji-j = Pj. Let g^ = g{T^ U , £^, P^)) denote the graph obtained. Note that X^ = X and 
= J \ jTieaf, and all the leaves of g^ are in X. Note also that since g^ is a tree, it contains 
at least two leaves unless its maximum degree equals 1. Let denote the collection of nonzero 
elements of T thus far defined. 

Given G^ = g{T^ U ,£^, (a^, P^)), for each A: = 1, 2,3,...,/ — 1, we perform the following: 

(i) Choose any leaf i G and set ifij^ = Oj and 7r(z) = k. Replace g^ with {g^Y Let 
^k+i 

(ii) For {g^^^ obtained in (i), perform the server leaf elimination as described above, and 
denote the resulting graph by g^~^^., and by denote the collection of nonzero elements 
of 4' thus far defined. 
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At step / — I, the resulting graph has a maximum degree of zero, where = {?} is a singleton 
and is empty and 'h contains exactly I + J — 1 non-zero elements. We set 7 r{i) = I. 

Remark 4.1. We remark that in the first step of server leaf elimination, all leaves in J' are removed 
while in each cnstomer leaf elimination, only one leaf in X (if more than one) is removed. Thus, 
exactly I steps of customer leaf elimination are conducted in the algorithm. The input of the 
algorithm is a tree Q with the vertices XVJ J^ the edges £ and the indices (a, j3). The outpnt of the 
algorithm is the matrix 'k = 'I'(a,/3)—the unique solution to the linear map G defined in (I2.18jl . 
and the permutation of the leaves X which tracks the order of the leaves being eliminated, that is, 
for each k = 1,2,...,/, 7 r(f) = k for some i £ X. Note that the permutation vr may not be unique, 
but the matrix T is unique for a given tree Q. The elements of the matrix T determine the drift 
b{x,u) = b{x,{u‘^,u^)) by (I2.23h . It is shown in the lemma below that the nonzero elements of 
T are linear functions of (a,/3), which provides an important insight on the structure of the drift 
b{x,u)] see Lemma (4^ 

Lemma 4.2. Let n denote the permutation ofX defined in the leaf elimination algorithm, and 7r~^ 
denote its inverse. For each k € X, 

(a) the elements of the matrix are functions of 

{*^7r“l(l) : ■ ■ ■ 1 ®7r“l(fc—1)5 P} ) 

(b) the set 

{ifij : i = Tr-^{l),... ,TT-^{k), j € j] 

and the set of nonzero elements of rows 7r“^(l),... ,7r“^(/) o/T are equal; 

(c) there exists a linear function such that 

*^7r~l(fc) ®7r“l(fc) (^7r“l (1) 5 • • • ) ^■n~^ {k—1 )! /^) ■ 

Proof. This is evident from the incremental definition of T in the algorithm. □ 

Lemma 4.3. The drift b{x,u) = b{x, {u^,u^)) in the limiting diffusion X in (j2.22jl can be expressed 
as 

b{x,u) = —Bi{x — {e-x)^u^) + {e-x)~B 2 U^ — {e-x)~^Fu‘^ + i, (4.1) 

where Bi is a lower-diagonal I x I matrix with positive diagonal elements, B 2 is an I x J matrix 
and F = diag{ 7 i,... , 77 }. 


Proof. We perform the leaf elimination algorithm and reorder the indices in X according to the 
permutation vr. Thns, leaf i G X is eliminated in step i of the customer leaf elimination. Let ji G J/ 
denote the nnique node corresponding to i G X, when i is eliminated as a leaf in step i of the 
algorithm. It is important to note that, with respect to the reordered indices, the matrix G^{x) 
(see Remark El]) takes the following form 



Xi + Gij^ (xi,..., Xi-i) for j = ji , 

< Gij{xi ,..., Xi-i) for f ~ j , j ji, 

0 otherwise, 

V ' 


where each Gij is a linear function of its arguments. As a result, by Lemma [4.21 the drift takes the 
form 


bi{x,u) = -pLij^Xi + bi{xl,...,Xi-l) + Fi[{e■x)'^u'',{e■x) u"") - ^i{e ■ x)^u1 + li. (4.2) 

Two things are important to note: (a) Xj is a linear function, and (b) pLij^ > 0 (since i ji)- 
Let b denote the vector field 

bi{x) := - 


RijiXi + bfixi,.. .,Xi-i). 


(4.3) 









ERGODIC DIFFUSION CONTROL OF MULTICLASS MULTI-POOL NETWORKS 


23 


Then 6 is a linear vector field corresponding to a lower-diagonal matrix with negative diagonal 
elements, and this is denoted by —Bi. The form of the drift in (14.Ill then readily follows by the 
leaf elimination algorithm and (|2.23p . □ 

Remark 4.2. By the representation of the drift b{x,u) in (|4.ip . the limiting diffusion X can be 
classified as a piecewise-linear controlled diffusion as discussed in Section 3.3 of [T]. The difference 
of the drift b(x,u) from that in [T] lies in two aspects: (i) there is an additional term (e • x)~B 2 U^, 
and (ii) Bi may not be an M-Matrix (see, e.g., the Bi matrices in the W model and the model in 
Example 14.41 below). 


4.2. Examples. In this section, we provide several examples to illustrate the leaf elimination 
algorithm, including the classical “N”, “M”, “W” models and the non-standard models that cannot 
be solved in UM- Note that in Assumption 3 of [ 6 ] (and in Theorem 1 of [5]), it is required that 
either of the following conditions holds: (i) the service rates fiij are either class or pool dependent, 
and 7 j = 0 for all i G I] (ii) the tree G is of diameter 3 at most and in addition, 7 * < jiij for each 
i ^ j m G- We do not impose any of these conditions in asserting Hypotheses 13.11 and 13.21 later in 
Section 14.31 


Example 4.1 (The “N” model). Let X = {1,2}, J' 
matrix T takes the form 'I'(a,/3) = 


= (1, 2} and X = {1 ~ 1,1 ~ 2, 2 ~ 2}. The 
and the permutation vr satisfies = k 


for A: = 1,2. The matrices Bi and B 2 in the drift b{x,u) are Bi = diag{/ii 2 ,// 22 } and B 2 = 
diag{/iii - //i 2 , 0 }. 


Remark 4.3. Recall G^{x) in Remark 12.11 Applying the leaf elimination algorithm, there may be 
more than one realizations of G^{x). For example in the ‘N’ network, the solution can be expressed 


as 4'(a, /S) = 


Pi 

0 


ai - Pi 
02 


, or '!'(«, /3) = 


Pi 

0 


P 2 — 02 
02 


and these give different answers when 


u = 0. It depends on the permutation order in the implementation of elimination, i.e., which pair 
of nodes is eliminated last. 


Example 4.2 (The “W” model). Let X = {1, 2, 3}, J = {1, 2} and X = {1 ~ 1, 2 ~ 1, 2 ~ 2, 3 ~ 2}. 
Following the algorithm, we obtain that the matrix 'k takes the form 


^{a,P) 


ai 0 

Pi -ai 02 - {Pi - oi) , 

0 03 


and the permutation vr satisfies vr ^{k) = k for k = 1,2,3. The matrices Bi and B 2 in the drift 
b{x, u) are 



k -11 

0 

0 



0 

O' 

Bi = 

/^21 + k-22 

k-22 

0 

and 

B 2 = 

/^21 — /^22 

0 


0 

0 

/^32_ 



0 

0 


Example 4.3 (The “M” model). Let X = {1, 2}, J = {1, 2, 3}, and X = {1 ~ 1,1 ~ 2, 2 ~ 2, 2 ~ 
3}. The matrix ^ takes the form 


4'(o,/3) 


'Pi ai- Pi O' 
0 a2 — Ps Ps 


and the permutation vr satisfies vr ^{k) 
are 

Bi = diag{/ii2,/U22} 


= k for k = 1,2. The matrices Bi and B 2 in the drift b{x, u) 


and B 2 


1^11 — k -12 0 0 

0 0 /V23 —1^22, 
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Example 4.4. Let X = {1, 2,3,4}, J = {1, 2,3} and S = {1 ^ 
We obtain 


ai 0 0 


1,2 




1,2~2,2~3,3~3,4~3}. 


4'(a,/3) 


Pi-ai 132 (a2 -/32) - (/3i - ai) 

0 0 as 

0 0 a4 


and the permutation vr satisfies tt ^(/c) = k for k = 1,2,3,4. The matrices Bi and B 2 in the drift 
6(x, u) are 


Ain 

0 

0 

0 ■ 



0 

0 

o' 

— Ai21 + Ai23 

0 

Ai23 

0 

0 

Ai33 

0 

0 

and 

B 2 = 

— Ai21 — T 23 

0 

-Ai23 

0 

0 

0 

0 

0 

0 

Ai43_ 



0 

0 

0 


4.3. Verification of Hypotheses 13.11 and 13.21 In this section we show that the controlled dif¬ 
fusions X in (I2.22P for the multiclass multi-pool networks satisfy Hypotheses 13.11 and 13.21 

Theorem 4.1. For the unconstrained ergodic control problem (I2.28p under a running cost r in 
(j2.26p with strictly positive vectors ^ and C,, Hypothesis \3.1\ holds for fC = fCs defined by 

)Cs := {x G : |e • x| > (5|x|} (4.4) 

for some <5 > 0 small enough, and for a function h{x) := C\x\'^ with some positive C. 

Proof. Recall the form of the drift b{x, u) in (14.Ih in Lemma 14.31 The set fCs in (14.4h is an open 
convex cone, and the running cost function r{x,u) = r(x, {u‘^,u^)) in (I2.26h is inf-compact on ICs- 
Dehne V G C^(R^) by V(x) := (x'^Qx)""^^ for |x| > 1, where m is as given in (j2.26p . and the matrix 
Q is a diagonal matrix satisfying x'^{QBi -(- BJQ)x > 8 |xp. This is always possible, since —Bi is 
a Hurwitz lower diagonal matrix. Then we have 

b{x,u)-VV{x) = £-VV{x)-'^{x'^Qxp^-^x'^{QBi +bJQ)x 

+ m{x'^Qx)”"''^~^Qx[{Bi — r)(e • x)^u‘^ + B 2 {e ■ x)~u^) 

< m{£^Qx){x'^— m{x'^QxY"^'^~^ (yA\x\^ — CilxHe • x|) 

for some positive constant Ci. Choosing 5 = we obtain 

b{x,u)-W{x) < C 2 — rn{x'^Qx)”'^^~^\x\‘^ VxG/C^, 

for some positive constant 6 * 2 . Similarly on the set JCs Id {|x| > 1}, we can obtain the following 
inequality 

b{x,u)-W{x) < C'i{l + \e ■ x\^) y X € JCs , 
for some positive constant C 3 > 0. Combining the above and rescaling V, we obtain 

£“V(x) < l-C4|xri,c^(x) + C'5|e-xriyc,(x), x G R^ 
for some positive constants C 4 and C 5 . Thus Hypothesis 13.II is satished. □ 

Remark 4.4. It follows by Theorem 14.II that Lemma [3.31 holds for the ergodic control problem with 
constraints in (|2.30l) - (|2.31l) under a running cost tq as in (I2.26P with ^ = 0. 

Theorem 4.2. Suppose that the vector 7 is not identically zero. There exists a constant Markov 
control u = {yC,hP') G U which is stable and has the following property: For any m>l there exists 
a Lyapunov function V of the form V(x) = {x'^Qx)'^^^ for a diagonal positive matrix Q, and positive 
constants kq and ki such that 

£“V(x) < Ko — KiV{x) VxGR^. 


( 4 . 5 ) 
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As a result, the controlled process under u is geometrically ergodic, and its invariant probability 
distribution has all moments finite. 


Proof. Let z G X be such that 7 j > 0. At each step of the algorithm the graph has at least two 
leaves in X, unless it has maximum degree zero. We eliminate the leaves in X sequentially until we 
end up with a graph consisting only of the edge (z, j). Then we set zz? = zzl = 1. This defines zz'^ 
and zz®. It is clear that zz = (zz^,zz^) G U. Note also that in the new ordering of the indices (replace 
with the permutation vr) we have i = I and and we can also let j = J. 

By construction (see also proof of Lemma 14.,lb , the drift takes the form 


bi{x,uo) 


bfix), ifz</, 

bi{xi,..., xi-i) - fiijxi - ( 7 / - fiij) (e • x)+ +£i, if z = /, 


where b is as in (|4.3p . Note that the term (e • x) does not appear in hi{x,ufi). The result follows 
by the lower-diagonal structure of the drift. □ 


Remark 4.5. It is well known [21 Lemma 2.5.5] that (|4.5p implies that 

E“[V(At)] < — + Vix)e-^^\ VxGM^ Vt > 0. (4.6) 

Kl 

4.4. Special cases. In the unconstrained control problems, we have assumed that the running cost 
function r(x,u) takes the form in (I2.26|) . where both the vectors ^ and f are positive. However, if 
we were to select C = 0 (thus penalizing only the queue), then in order to apply the framework in 
Section 13.11 we need to verify Hypothesis 13.11 for a cone of the form 

}Cs^+ := {x G : e • x > djxj} , (4.7) 

for some <5 > 0. Hypothesis 13.11 relative to a cone JCs^+ implies that, for some a > 0, we have 

Jy[{e-x)~] < Kj^[(e-x)+] VuGHsm- (4.8) 

In other words, if under some Markov control the average queue length is finite, then so is the 
average idle time. 

Consider the “W” model in Example 14.21 When e • x < 0, the drift is 


b{x, zz) 


/zii 0 0 

/Z2l(l + zzf) -h h2iul + ia22U2 h2iul + H22U2 


0 0 /Z 32 

We leave it to the reader to verify that Hypothesis 13.11 holds relative to a cone with a function 
V of the form V(x) = {x'^Qx)'^^^. The same holds for the “N” model, and the model in Example 14.41 
However for the “M” model, when e • x < 0, the drift takes the form 


b{x,u) = — 


X + £. 


/Zi2(l - Zzf) +/ZllZzf (/Zll - /Zl2)zzf 

{h23 — R22 )uI /Z22(1 “ wf) + Ai23^ 

Then it does not seem possible to satisfy Hypothesis |3T] relative to the cone unless restrictions 
on the parameters are imposed, for example, if the service rates for each class do not differ much 
among the servers. We leave it to the reader to verify that, provided 


ll^ll — /^12| V I/Z 23 —/Z22I < I (/^12 A/Z22), 

Hypothesis 13.II holds relative to the cone with Q equal to the identity matrix. An important 
implication from this example is that the ergodic control problem may not be well posed if only 
the queueing cost is minimized without penalizing the idleness either by including it in the running 
cost, or by imposing constraints in the form of (I2.31h . 

We present two results concerning special networks. 
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Corollary 4.1. Consider the ergodic eontrol problem in (I2.28P with X in (I2.22p and r{x,u) in 
(I2.26P with ( = 0. For any m > 1, there exist positive constants 5, 5, and k, and a positive definite 

Q G such that, if the service rates satisfy 

max - Pik\ < d max {pij} , 

then with V{x) = {x'^Qx)”^/'^ and in (14.71) we have 

£“V(x) < k- |xr Vx G , Vu G U. 

Proof. By ()2.18p . (I2.20p and ()2.23p . if Hij = pik = fi for alH G X and j, k £ then bi(x, u) = —fixi 

when e • X < 0, for all i G X. The result then follows by continuity. □ 

Corollary 4.2. Suppose there exists at most one i G X such that \ J{i)\ > 1. Then the conclusions 
of Corollary \4.1\ hold. 

Proof. The proof follows by a straightforward application of the leaf elimination algorithm. □ 

Remark 4.6. Consider the single-class multi-pool network (inverted “V” model). This model has 
been studied in Hi. The service rates are pool-dependent, pj for j G J. The limiting diffusion 
X is one-dimensional. It is easy to see from (12.231) that 

6 (x, u) = x~ pjUj — ■yx'^ -|- £ 
j&J 

= -yx + x~ pjUj - 1 - 7 ^ + i. 

It can be easily verified that the controlled diffusion X for this model not only satishes Hypoth¬ 
esis ED relative to but it is positive recurrent under any Markov control, and the set of 

invariant probability distributions corresponding to stationary Markov controls is tight. 

Remark 4.7. Consider the multiclass multi-pool networks with class-dependent service rates, that 
is, pij = Pi for all j £ J(i) and i £X. In the leaf elimination algorithm, the sum of of the elements 
of row i of the matrix 'I'(q:, /?) is equal to ctj, for each z G X. Thus, by (j2.23p . we have 

bi{x,u) = bi{x,{u‘^,u'')) =-pi{xi - {e ■ x^Ui) - yfie ■ x^uf + ii VzGX. 

This drift is independent of u^, and has the same form as the piecewise linear drift studied in 
the multiclass single-pool model in [T]. Thus, the controlled diffusion X for this model satisfies 
Hypothesis 13.11 relative to Xs^^. Also Hypothesis 13.21 holds for general running cost functions that 
are continuous, locally Lipschitz and have at most polynomial growth, as shown in [T]. 

5. Characterization of Optimality 

In this section, we characterize the optimal controls via the HJB equations associated with the 
ergodic control problems for the limiting diffusions. 

5.1. The discounted control problem. The discounted control problem for the multiclass multi¬ 
pool network has been studied in [5]. The results strongly depend on estimates on moments of the 
controlled process that are subexponential in the time variable. We note here that the discounted 
infinite horizon control problem is always solvable for the multiclass multi-pool queueing network 
at the diffusion scale, without requiring any additional hypotheses (compare with the assumptions 
in Theorem 1 of IS])- Let 51 : X U —)• M+ be a continuous function, which is locally Lipschitz in 

X uniformly in u, and has at most polynomial growth. For 9 > 0, dehne 

39{x-,U) := 


/ 

L ./0 




giXs,Us)ds 


(5.1) 
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It is immediate by (I4.6p that 3eix]u) < oo and that it inherits a polynomial growth from g. 
Therefore inffygu 3e{x;U) < oo. It is fairly standard then to show (see Section 3.5.2 in [2]) that 
Ve(x) := 3eix]U) is the minimal nonnegative solution in of the discounted HJB 

equation 

i trace(SS'^VVe(x))+//(x,VI/ 0 ) = 0 I 4 (x), x £ 

where 

H{x,p) := min \b{x,u) ■ p + g{x,u)] . (5.2) 

Moreover, a stationary Markov control v is optimal for the criterion in (15.ip if and only if it satisfies 
b[x,v{x)) ■'VVe{x) + g[x,v{x)) = H[x,'VVo{x)) a.e. in . 


5.2. The HJB for the unconstrained problem. The ergodic control problem for the limiting 
diffusion falls under the general framework in [T]. We state the results for the existence of an 
optimal stationary Markov control, and the existence and characterization of the HJB equation. 

Recall the dehnition of Jx,u['>'] and g*{x) in (I2.27p - (l2.28p . and recall from Section [3^1 that if 
V G ilssM) then Jx^vix] does not depend on x and is denoted by Consequently, if the ergodic 

control problem is well posed, then q*{x) does not depend on x. We have the following theorem. 

Theorem 5.1. There exists a stationary Markov control v G ilsSM that is optimal, i.e., it satisfies 
= Q*- 

Proof. Recall that Hypothesis 13.11 is satished with /i(x) := C\xY^ for some constant C > 0, as in 
the proof of Theorem 14.11 It is rather routine to verify that (j3.6l) holds for an inf-compact function 
/i ~ \x\^. The result then follows from Theorem 3.2 in [T]. □ 


We next state the characterization of the optimal solution via the associated HJB equations. 


Theorem 5.2. For the ergodic control problem of the limiting diffusion in ()2.28p . the following 
hold: 

(i) There exists a unique solution V G C^(M^) n Odxl™), satisfying H(0) = 0, to the associated 
HJB equation: 

min [£“H(x)-I-r(x,tt)] = g* . (5.3) 




The positive part ofV grows no faster than \x\^, and its negative part is in o(|x|”^). 

(ii) A stationary Markov control v is optimal if and only if it satisfies 

H(x,VH(x)) = 6(x,u(x)) • VH(x)-|-r(x,x(x)) a.e. inR^ , 

where H is defined in ( 1521 ). 

(iii) The function V has the stochastic representation 


(5.4) 


V(x) = lim inf 


El 


[ {r{Xs,v{Xs)) - g*) ds 
Jo 


= lim El 
5\o 


[ {r[Xs,v^{Xs)) - g*) ds 
Jo 


for any v G ilsM that satisfies (|5.4I) . where u* is the optimal Markov control satisfying (15.4p . 

Proof. The existence of a solution V to the HJB (15.31) follows from Theorem 3.4 in pQ. It is 
facilitated by defining a running cost function r£(x, u) := r(x, u) + sh{x, u) for e > 0 , and studying 
the corresponding ergodic control problem. Uniqueness of the solution V follows from Theorem 3.5 
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The claim that the positive part of V grows no faster than \x\^ follows from Theorems 4.1 and 
4.2 in [T], and the claim that its negative part is in o(|x|”*) follows from Lemma 3.10 in [T]. 

Parts (ii)-(iii) follow from Theorem 3.4 in [T]. □ 


For uniqueness of solutions to HJB, see [H Theorem 3.5]. 

The HJB equation in (I5.3jl can be also obtained via the traditional vanishing discount approach. 
For a > 0 we define 


Va(x) := inf E; 

f/GU ' 


—at 


r{Xt,Ut)dt 


The following result follows directly from Theorem 3.6 of [T]. 


(5.5) 


Theorem 5.3. Let F* and g* be as in Theorem 15.^1 and let Va be as in ([53]). The function 
Va — Va{0) converges, as a \ 0, to F*, uniformly on compact subsets o/M^. Moreover, aFa(O) —)• g*, 
as q; \ 0 . 


The result that follows concerns the approximation technique via spatial truncations of the 
control. For more details, including the properties of the associated approximating HJB equations 
we refer the reader to [H Section 4]. 

Theorem 5.4. Let u G U satisfy ()4.5p . There exists a sequenee {v^ G Hssm : A: G N} such that 
each Vk agrees with u on B^, and 

Jvulr] - - 1 

K —>-00 


Proof. This follows by Theorems 4.1 and 4.2 in [T], using the fact that h ~ V ~ |x|”^. □ 

Since U is convex, and r as defined in (|2.26p is convex in u, we have the following. 


Theorem 5.5. Let h G U satisfy (14.5p . Then, for any given s > 0, there exists an R > 0 and an 
e-optimal continuous precise control Ve G ilssM which is equal to u on Bf^. In other words, if 
is the ergodic occupation measure corresponding to Ve, then 


7t, 


Ve 


(r) = 


L 


R'^xU 


r{x,u) ny^{dx,du) < g* + s . 


Proof. Let / : U —>• [0,1] be some strictly convex continuous function, and define r^{x,u) := 
r{x,u) + ^f{u), for e > 0. Let be the optimal value of the ergodic problem with running cost 
r^. It is clear that gt < g* + ^. 

Let vq G ilssM be the constant control which is equal to u, and for each i? G N, let b^{x,u) be 
as defined in (j3.9p and analogously define rf{x,u) as in (|3.10p relative to r^. Let denote the 
controlled extended generator of the diffusion with drift b^ in (I3.11h . Consider the associated HJB 
equation 

min [£^Fij(x) + r^(x, u)] = g{£,R). (5.6) 

uGU 

Since u i-A [b^{x, u)-Vr + r^{x, u)] is strictly convex in tt for x G Br, and Lipschitz in x, it follows 
that there is a (unique) continuous selector from the minimizer in (|5.6I) . By Theorem 15.41 fsee 
also Theorems 4.1 and 4.2 in [T]) we can select R large enough so that 

g{s,R) < £>* + I • (5-7) 

Next we modify so as to make it continuous on W^. Let {xk : A; G N} be a sequence of 
cutoff functions such that Xk £ [0,1], Xfe = 0 on B‘i_i, and Xfc = 1 on Bp_ 2 . For R fixed and 

satisfying (15.71) . define the sequence of controls Vk,eix) ■= Xk{x)ve,R{x) + (1 — Xk{x))vo{x), and let 

7 tfc denote the associated sequence of ergodic occupation measures. It is evident that 

in the topology of Markov controls [21 Section 2.4]. Moreover, the sequence of measures is 
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tight, and therefore converges as A; —)• oo to the ergodic occupation measure Tr^ corresponding to 
Ve,R [H Lemma 3.2.6]. Since is uniformly integrable with respect to the sequence we have 


/ re{x,u)nk{dx,du) - 

J-Rd-xV k^oo 

Combining this with the earlier estimates finishes the proof. 


□ 


5.3. The HJB for the constrained problem. As seen from Theorems 13. II and 13.21 the dynamic 
programming formulation of the problem with constraints in (l2.3Up - (l2.31|) follows in exactly the 
same manner as the unconstrained problem. Also, Theorems 13.31 and 13.41 apply. 

We next state the analogous results of Theorems 15.41 and 15.51 for the constrained problem. 

Theorem 5.6. Let u £ U satisfy (14.51) . Suppose that 6 G (0, 00 )*^ is feasible. Then there exist 
ko gN and a sequence {vk G Ussm : A: G N} such that for each k > ko, Vk is equal to u on B'j^ and 


JvM - -^ Qc 

/c—)-oo 

Proof. This follows from Theorem 13.41 


inf 

7tG3<(6) 


7t(ro). 


□ 


Since rj{x, u) defined in (I2.29P is convex in u for j = 0,1,..., J, we have the following. 


Theorem 5.7. Let ft G U satisfy (14.5p . Suppose that 5 G (0,oo)'^ is feasible. Then for any given 
e > 0, there exists Rq > 0 and a family continuous precise controls Vs^r G UssMj R > Rq satisfying 
the following: 

(i) Each Ve,R is equal to u on Bf^. 

(ii) The corresponding ergodic occupation measures ^ satisfy 

< 9* +£ yR>Ro, (5.8a) 

sup ny^jiirj) < jGj- (5.8b) 

R>Ro 

Proof. By Lemma 13.41 for all sufficiently small e > 0, there exist 6^ < 8j, j G such that 5^ is 
feasible and 


inf 7r(ro) < inf 7T(ro) + 

7rGM(5'=) 7tGM(6) 


£ 

4 • 


Let 


u) := 56,A (a;, ^^) + § f{u ), A G M+ , 

where e > 0, 55 ^X 1 is as in Definition 13.41 and / : U —)• [0,1] is some strictly convex continuous 
function. Let vq G Ussm be the constant control which is equal to ft, and for each ii G N, let 
6^(x, u) be as defined in (13.9p . Recall the definition of S(ii) and 9f(6; R) in the paragraph preceding 
Theorem 13.31 By Theorem 13.41 there exists G such that 


inf 

7tGS(R) 


<9leX^) 


inf 

7tGM(5GR) 


'^{'^0 + I /) , 


and (I3.28jl holds, and moreover, ii > 0 can be selected large enough so that 


inf 

TIG M(6‘^;R) 


+ I /) 


Combining these estimates we obtain 


inf 

TtGS(R) 




< inf 
tig W(6«) 

< inf 
tig M(6£) 


< inf 

tiGM(5) 


'^{^0 + I /) + 


7t(ro) + I . 


7t(ro) + T ■ 


£ 

4 


By strict convexity there exists a (unique) continuous selector Ve,R from the minimizer in (|3.28p . 
Using a cutoff function x as in the proof of Theorem l5.5l and redefining completes the argument. □ 
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5.4. Fair allocation of idleness. There is one special case of the ergodic problem under con¬ 
straints which is worth investigating further. Let 

:= {z G : e • 2 : = 1} . 

Consider the following assumption. 

Assumption 5.1. Hypothesis 13.11 holds relative to a cone }Cs^+ in (14.41) . and for every G S'^ 
there exists a stationary Markov control v{x) = such that Jv[ro] < 00 . 


Examples of networks that Assumption 15.11 holds were discussed in Section 14.21 In particular, it 
holds for the “W” network, the network in Example 14.41 and in general under the hypotheses of 
Corollaries 10 and [Ql 

Let ro(x,tt) be as defined in ()2.26p with C = 0) 

rj{x,u) := (e • x)~Uj , j £ J , 

Let 0 be an interior point of 5“^, i.e., Qj > 0 for all j G and consider the problem with constraints 
given by 

g* = inf J„[ro] (5.9) 

V e Hssm 

J 

subject to Jv[rj] = Qj'^Jv[rk], j = (5.10) 

k=l 

The constraints in (I5.10p impose fairness on idleness. In terms of ergodic occupation measures, the 
problem takes the form 

Q* = inf 7T(ro) (5.11) 

7teg 


subject to 7T(rj) = Qj''^^n{rk), j = 


(5.12) 


k=l 


Following the proof of Lemma 13.31 using (|4.8I) and Assumption 15.11 we deduce that the infimum in 
(I5.1ip - ()5.12p is finite, and is attained at some tt* G S- Dehne 

J-i J 

L{n, A) := 7t(ro) + ^ Aj (n{rj) - Qj ^ 7t( 

We have the following theorem. 


k=l 


Theorem 5.8. Let Assumption 15.11 hold. Then for any 0 in the interior of S'^ there exists a 
V* G ilssM which is optimal for the ergodic cost problem with constraints in (|5.9l) - (l5.10p . Moreover, 
there exists A* G such that 

g*, = inf L(7r,A*), 

Ties 

and V* can be selected to be a precise control. 


Proof. The proof is analogous to the one in Lemma 13.51 It suffices to show that the constraint 
is linear and feasible (see also [201 Problem 7, p. 236]). Let S := {tt G S : 7T(ro) < 00 }. By the 
convexity of the set of ergodic occupation measures, it follows that S is a convex set. Consider the 
map F : S —^ given by 

J 

Fj{n) := n{rj) - Bj ^ n{rk), j = 1,... , J - 1. 

k=l 

The constraints in (I5.12p can be written as F{n) = 0 and therefore are linear. 
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We claim that 0 is an interior point of F{S)- Indeed, since 0 be an interior point of for 
each j G {1,..., J — 1} we may select u® G 5“^ such that Uj = dj for j G {1,..., J — 1} \ {j}, and 
Uj > 0j. By Assumption 15.11 there exists v G ilssMj of the form v = such that G S- It 

is clear that Fj(Tty) = 0 for j / j, and Fj{ny) > 0. Repeating the same argument with u| < 0j we 
obtain tt^ G S such that Fj{ny) = 0 for j / j, and Fj{Tty) < 0. Thus we can construct a collection 
So = {tti, ..., 7T2j_ 2} of elements of S such that 0 is an interior point of the convex hull of -F(So)- 
This proves the claim, and the theorem. □ 

Remark 5.1. Theorem 15.81 remains of course valid if fewer than J — 1 constraints, or no constraints 
at all are imposed, in which case the assumptions can be weakened. For example, in the case of 
no constraints, we only require that Hypothesis 13.11 holds relative to a cone in (14.4p . and the 
results reduce to those of Theorem 15.21 

Also, the dynamic programming counterpart of Theorem 15.81 is completely analogous to Theo¬ 
rem [3Tl and the conclusions of Theorems 15.61 and 15.71 hold. 

6. Conclusion 

We have developed a new framework to study the (unconstrained and constrained) ergodic 
diffusion control problems for Markovian multiclass multi-pool networks in the Halfin-Whitt regime. 
The explicit representation for the drift of the limiting controlled diffusions, resulting from the 
recursive leaf elimination algorithm of tree bipartite networks, plays a crucial role in establishing 
the needed positive recurrence properties of the limiting diffusions. These results are relevant to 
the recent study of the stability/recurrence properties of the multiclass multi-pool networks in 
the Halhn-Whitt regime under certain classes of control policies |22H25j . The stability/recurrence 
properties for general multiclass multi-pool networks under other scheduling policies remain open. 
It is important to note that our approach to ergodic control of these networks does not, a priori, 
rely on any uniform stability properties of the networks. 

We did not include in this paper any asymptotic optimality results of the control policies con¬ 
structed from the HJB equation in the Halfin-Whitt regime. We can establish the lower bound 
following the method in [1] for the “V” model, albeit with some important differences in technical 
details. The upper bound is more challenging. What is missing here, is a result analogous to 
Lemma 5.1 in [T]. Hence we leave asymptotic optimality as the subject of a future paper. 

The results in this paper may be useful to study other diffusion control problems of multiclass 
multi-pool networks in the Halhn-Whitt regime. The methodology developed for the ergodic control 
of diffusions for such networks may be applied to study other classes of stochastic networks; for 
example, it remains to study ergodic control problems for multiclass multi-pool networks that do 
not have a tree structure and/or have feedback. This class of ergodic control problems of diffusions 
may also be of independent interest to the ergodic control literature. It would be interesting to 
study numerical algorithms, such as the policy or value iteration schemes, for this class of models. 
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